Sampling Circulant Matrix Approach: A Comparison of Recent Kernel Matrix Approximation Techniques in Ridge Kernel Regression

As part of a survey of state-of-the-art kernel approximation algorithms, we present a new sampling algorithm for circulant matrix construction to perform fast kernel matrix inversion in kernel ridge regression, comparing theoretical and experimental performance of that of multilevel circulant kernel approximation, incomplete Cholesky decomposition, and random features, all recent advances in the literature. In particular, the new circulant approach rivals the remaining three algorithms in accuracy, executing in time complexity of mixed competitiveness, warranting further study. 1 Survey of the Problem: Ridge Regression Ridge regression, also known as Tychonoff regularization, appears as early as [1], though [2] standardized the approach in statistics. Formally, given a D × N data matrix X and corresponding vector of labels y, ridge regression estimates f(x) = y with f̂(x) = wx, where w is a weight vector minimizing ||wX − y|| + ||wΓD||, (1) where Γ is a regularization matrix. The exact solution is ŵ = (XX + ΓDΓ T D) −1Xy. (2) Typically, we choose Γ to be a diagonal matrix with positive entries. Herein, we replace ΓDΓD with λID, where ID is the identity matrix.